J 



Europaisches Patentamt 
European Patent Office 
Office europeen des brevets 



@ Application number: 91311251.2 
@ Date of filing : 03.12.91 



© Publication number : 0 489 576 A2 
EUROPEAN PATENT APPLICATION 

@ int. ci. 5 : G06F 3/023 



(So) Priority : 04.12.90 US 622821 

@ Date of publication of application : 
10.06.92 Bulletin 92/24 

(§4) Designated Contracting States : 
DE FR GB 

@ Applicant: SONY CORPORATION OF 
AMERICA 
Sony Drive 

Park Ridge, New Jersey 07656 (US) 



(72) Inventor: Mackay, Michael T. 
100 Bluebell Place 
Vallejo, California 94591 (US) 
Inventor : Berger, Robert J. 
935 College Avenue 
Merlo Park, California 94025 (US) 
Inventor : Duffy, Robert 
166 Midwick Drive 
Milpitas, California 95035 (US) 
Inventor : Langford, Ted Ellis 
348 Rivercreek Drive 
Fremont, California 94536 (US) 

@ Representative : Cotter, Ivan John et al 
D. YOUNG & CO. 10 Staple Inn 
London WC1V 7RD (GB) 



0_ 

UJ 



(54) Computer controlled display apparatus and method. 

(57) Apparatus and methods are described to pro- 
vide a multi-dimensional user interface for use 
in audio visual production. A display system 
including a central processing unit (CPU) (22) is 
coupled through appropriate input/output (I/O) 
circuitry (32) to input devices, such as a 
keyboard (38), a digital pad (36) and/or a track 
ball (40) as well as a display device (50). The 
CPU is further coupled to a hard disk drive (30) 
for the storage of programs and data, and is 
also coupled to a network through which the 
CPU may communicate with a variety of system 
resource devices such as editors, music synth- 
esizers (55), graphics generators, scheduling 
resources, audio enhancement resources, etc. 
A user viewing the interface on the display may 
utilize one of the input devices, such as by way 
of example, the keyboard, to select, incorporate 
or otherwise integrate the various system re- 
sources to develop a unified multi-media pro- 
duction. The user interface includes a control 
frame (150) which in practice substantially fills 
all of the display screen of the display and is 
consistent for all user applications. The control 
frame is comprised of control panels (152, 190, 
170) which surround a variety of subwindows 
(200, 215) and acts as a consistent control area 
for all users of the interface. Once defined, 
elements may be selectively placed on an event 
horizon bar (220) in the control frame. The 
placement of an element on the event horizon 
results in the display of timing data for the 
element, relative to other elements on the event 
horizon. 
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This invention relates to apparatus and methods for displaying, manipulating, and selecting multi-media 
or computer stored information in a window environment of a computer display system. More particularly, this 
invention relates to an improved user interface to provide a unified operator interface for a wide range of systems 
which must be coordinated and monitored in a multi-media production system. 

5 Videotape editing environments have evolved from providing simple editing cuts to the incorporation of full 

featured graphics, film to tape and other processes to complete a video production. Consequently, computer 
controlled editing systems and integration methods have been used to incorporate and integrate various pro- 
duction media resources such as special effects, music, graphics or the like. However, due to the nature of film 
and video production, a variety of resources must be integrated, scheduled and coordinated with one another 

w to obtain a completed product. 

Historically, humans have interfaced with computers through a system of discrete commands which typi- 
cally comprise a combination of both text and mathematical symbolic characters. Examples of such systems 
are numerous and include the programming languages of Fortran, Algol, Basic, etc., which transform a given 
set of user commands into machine executable "object" code. However, the ease with which a user becomes 

15 proficient in programming or interacting with a computer-based system is generally a function of how close the 
system models the logical thought of the user himself. If the user is able to enter commands in the order in 
which he would find most logically appropriate, rather than having to transpose his desired command into the 
code of a programming language, greater user efficiency in using the system is achieved. 

A number of systems which have been developed to reduce the learning or acclimatization period which 

20 a user must go through to become proficient in the interaction with the computer system are referred to as 
"object oriented systems". A common object oriented interface approach utilizes multiple "windows" displayed 
on a cathode ray tube (CRT) in which combinations of text and graphics are used to convey information. Each 
window may take the form of an object such as a file folder, different operating environment, pages or layered 
bit maps to provide a separate display of video information in independent screen regions. (See, for example, 

25 Robson "Object Oriented Software Systems", Byte, August, 1981; United States Patents Nos. 4,414,628, 
4,533,91 0, 4,450,442, 4,555,775 and 4,622,545; and L. Tesler , 'The Small Talk Environment" Byte, August 
1981, Volume 6, No. 8). 

The use of modern computer systems incorporating object oriented window environments may be applied 
to multi-media production methods, such as videotape editing, audio mixing, etc. However, one unique problem 

30 associated with multi-media production is the necessity to provide the ability for a diversity of media profes- 
sionals to collaborate and exchange project data in a consistent interface environment. By providing a consis- 
tent user interface, media professionals such as special effects engineers, animation specialists, music 
composers, and the like may provide both real time and non-real time input to exchange necessary project data, 
and effectively coordinate the production of the entire media work. Accordingly, one of the requirements of any 

35 common multi-media user interface is the ability to integrate multi-media types, and to provide the operator with 
the ability to manage large quantities of information in an understandable and efficient manner. The user inter- 
face must be intuitive and flexible to accommodate a variety of operator editing styles and personalities. For 
example, a music composer who thinks typically in terms of scores, notes and related music timing, should be 
able to work in that environment using a standard user interface, and not be required to work in terms of video 

40 time code or other non-music related external standards. Similarly, the film production director or special effects 
engineer should be able to utilize the user interface in a manner consistent with their work environment, which 
may, by way of example, be illustrated through the use of video time code signals (and not music). 

Viewed from one aspect this invention provides a computer controlled display apparatus including at least 
one central processing unit, said CPU being coupled to a display for displaying data, and user input means, 

45 said display system being further coupled to a plurality of system resources having defined attributes, said dis- 
play system comprising: 

means for generating a user interface for display by said display, said user interface including a display 
of representations of resources coupled to said display system with which a user interacts through said user 
input means, said representations of said resources being arranged in an N dimensional venue which may be 
50 viewed using said user interface from a plurality of view ports such that viewing said representations of said 
resources from different view ports results in the display of different attributes of said resources, said repre- 
sentations of said resources being arranged in said venue such that each of said resources is disposed relative 
to one another in time and space within said venue; and 

manipulation means coupled to said user input means for selectively positioning said representations 
55 of said resources within said venue. 

Apparatus and methods for generating a multi-dimensional user interface for use in audio visual production 
are provided. A display system including at least one central processing unit (CPU) may be coupled through 
appropriate input/output (I/O) circuitry to input devices, such as a keyboard, digital pad and/or track ball. The 
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CPU may be further coupled to a hard disk drive for the storage of programs and data, and may also be coupled 
to a network through which the CPU may communicate with a variety of system resource devices such as 
editors, music synthesizers, graphics generators, scheduling resources, audio enhancement resources, etc. 
The CPU is also coupled to a display device (for example, a CRT) on which the user interface is displayed to 

5 the user. A user viewing the interface on the display may utilize one of the input devices, such as by way of 
example, the keyboard, to select, incorporate or otherwise integrate the various system resources to develop 
a unified multi-media production. 

In preferred embodiments, the user interface includes a control frame which in practice substantially fills 
all of the display screen of the display. The control frame is comprised of control panels which surround a variety 

w of subwindows, and acts as a consistent control area for all users of the interface. The control frame may include 
a construction area for all users of the interface. The control frame may include a construction area which cor- 
responds typically to a front view port looking towards a three-dimensional element which is a representation 
of a resource. The control frame may further include a top view port which illustrates the time relationship be- 
tween the various resources in a "venue". Effectively, the control frame provides a two-dimensional window to 

15 selectively view a three-dimensional "element". 

In preferred embodiments, a user specifies mandatory and optional attributes which an element must have, 
and defines the element representing the resource within the construction area of the control frame. Once 
defined, the element may be selectively "dragged" down to an event horizon bar at which time, time data is 
displayed in the time view port of the control frame. Using the interface, elements may be created, edited, bun- 

20 died, integrated and rearranged along the event horizon. 

Viewed from another aspect the invention provides a resource control apparatus comprising: 
a display, 

user interface generation means for displaying on said display representations of a plurality of resources 
that may be coupled to said resource control apparatus, said representations being displayable by said user 
25 interface generation means from a plurality of view ports such that viewing said representations from different 
view ports results in the display of different attributes of said resources in a manner consistent with said rep- 
resentations being of N dimensional elements disposed within an N dimensional venue, and 

manipulation means responsive to a user input means for selectively positioning said representations within 
said venue. 

30 An embodiment of the invention will now be described, byway of example only, with reference to the accom- 

panying drawings in which: 

Figure 1 is a conceptual illustration of an integrated multi-media user interface for integrating and mani- 
pulating a variety of multi-media functions. 

Figure 2 is a functional block diagram showing one possible data processing system embodying this inven- 

35 tion. 

Figure 3 is a conceptual illustration of the use of venues to represent data sets of resources available to 
a user. 

Figure 4 conceptually illustrates the use of venues and view ports by the user interface. 
Figure 5 is a conceptual illustration of an element representing a resource, as viewed in three dimensions 
40 utilizing the user interface. 

Figure 6 is a front view of a user display system utilizing the multi-media user interface. 
Figure 7 is the same view as Figure 6 except that the element attributes window opened for venue selection. 
Figure 8 is the same view as Figure 7 further illustrating the user interface in the display of a plurality of 
elements and venues selected for the production of a multi-media work. 
45 Figure 9 is the same view as Figure 8 further illustrating the element attribute block opened and venue and 

resource selection. 

The detailed descriptions which follow are presented largely in terms of graphics interfaces, algorithms, 
and in symbolic representations of operations of data bits within a computer display system. These algorithmic 
descriptions and representations are the means used by those skilled in the data processing arts to most effec- 
50 tively convey the substance of their work to others skilled in the art. 

An algorithm is here, and generally, conceived to be a self-consistent sequence of steps leading to a des- 
ired result. These steps are those requiring physical manipulations of physical entities. Usually, though not 
necessarily, these entities take the form of electrical or magnetic signals capable of being stored, transferred, 
combined, compared, and otherwise manipulated. It proves convenient at times, principally for reasons of com- 
55 mon usage, to refer to these signals as bits, values, elements, symbols, icons, characters, terms, numbers, 
windows or the like. It should be borne in mind, however, that all of these similar terms are to be associated 
with the appropriate physical entities and are merely convenient labels applied to these entities. 

The manipulations performed are often referred to in terms, such as adding or comparing, displaying, etc. 
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which are commonly associated with mental operations performed by human operator. No such capability of 
a human operator is necessary, or desirable in most cases, in any of the operations described herein which 
perform part of the described embodiment of the present invention. In the present case, the operations are 
machine operations. Useful machines for performing such operations include general purpose digital compu- 
5 tens or other similar devices. In all cases, there should be borne in mind the distinction between the method 
operations of operating a computer and the method of computation itself. This invention relates to method steps 
for operating a computer graphics system and processing electrical or other physical signal representing physi- 
cal entities. 

This invention also relates to apparatus for performing these operations. This apparatus may be specially 
10 constructed for the required purposes, or it may comprise a general purpose computer selectively activated or 
reconfigured by a computer program stored in the computer. The algorithms and displays presented herein are 
not inherently related to any particular computer or other apparatus. Various general purpose machines may 
be used with programs in accordance with the teachings herein, or it may prove convenient to construct more 
specialized apparatus to perform the required method steps. The required structure for an embodiment of such 
15 a machine will appear from the description below. In addition, this invention is not described with reference to 
any particular programming language. It will be appreciated that a variety of programming languages may be 
used to implement this invention as described herein. 

The following detailed description will be divided into several sections. The first of these will treat a general 
system arrangement for generating computer graphics. Subsequent sections will deal with such aspects as the 
20 general conceptual definition of a "venue" and "resource", and the structure in operation of the multi-media user 
interface. 

In addition, in the following description, numerous specific details are set forth such as functional blocks 
representing data processing devices, window configurations, etc. in order to provide a thorough understanding 
of this invention. However, it may be obvious to one skilled in the art that this invention may be practised without 

25 these specific details. In other instances, well known circuits are structures are not described in detail in order 
not to obscure the present invention unnecessarily. 

Referring now to Figure 1, an embodiment of this invention is conceptually illustrated. As previously dis- 
cussed, there is provided an integrated user interface, such that media professionals may utilize a common 
interface and integrate various production processes such as animation, special affects, editing, mixing and 

30 production scheduling. Through the use of the user interface, a variety of resources may be accessed in the 
production of a feature film, videotape of the like. Unlike prior art systems in which separate special effects pro- 
duction facilities, live action facilities and music score editing facilities are required to work independently, and 
then be integrated at some later dated, there is provided a new editing facility to permit an operator to interact 
with each of the resources comprising a multi-media production and generate a final completed work. As will 

35 be described more fully below, this facility accesses, arranges and coordinates these various production resour- 
ces through the use of an integrated user interface. 

Referring now to Figure 2, one possible computer graphics system employing the teachings of this invention 
is shown. As illustrated, the graphics system includes a computer 20, which comprises six major components. 
The first of these is a central processing unit (CPU) 22 which is coupled to a memory 24. The CPU 22 and the 

40 memory 24 are further coupled to a user interface circuit 26 and a disk input/output (I/O) circuit 28, for com- 
municating with a hard disk drive 30 for mass storage of data. The computer 20 further includes a serial 
input/output (I/O) circuit 32 for communication with serial devices over line 34, such as by way of example a 
digitizing pad 36, a keyboard 38, and track ball input device 40. The computer system 20 further includes a 
network interface circuit 42, which is coupled to a network 44 and a gateway circuit 48, which permits the com- 

45 puter system 20 to communicate with other communication systems over telephone lines, optical fibres and 
the like. 

In addition, a display monitor 50 is illustrated which is used to display an integrated user interface, and is 
coupled to the user interface circuit 26. As also illustrated, a variety of resource devices such as a video tape 
recorder (VTR) 52, a music synthesizer 55, an audio tape recorder 60 and a production switcher 62 are coupled 
so to the network interface circuit 42 over the network 44, through device translators 64, 66, 68 and 70, respect- 
ively. 

As will be described more fully bellow, the arrangement illustrated in Figure 2 permits data from resources 
such as the VTR 52, the music synthesizer 55 and the audio tape recorder 60 to be coupled to the user interface. 
A user viewing the interface on the display 50 may utilize one of a variety of input devices, such as by way of 
55 example, the keyboard 30 or the track ball 40 to select, incorporate and otherwise integrate the various system 
resources to develop a unified multi-media production. 

It will be appreciated that the embodiment illustrated with reference to Figure 2 is only one possible embo- 
diment of many. For example, although only one computer 20 is shown, embodiments may include multiple 
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CPU's and/or computers coupled to the network 44. Each of these CPU's and/or computers coupled to the net- 
work 44 may execute separate programs, and support different versions of the user interface. Alternatively, it 
is contemplated that the use of multiple computers and/or CPU's may support a system of distributed comput- 
ing, wherein multiple versions of the user interface may be supported concurrently, each of the multiple user 

5 interfaces executing different processes but having access to a common data base of resources. As such, it 
will be noted that the embodiment shown in Figure 2 is a simplified embodiment for purposes of illustration, 
and is not meant to limit this invention's utility. 

Referring now to Figure 3, the described embodiment of this invention conceptually permits a variety of 
media resources such as special effects 90, editing sequencer 92 and video resources 95 to be viewed by a 

10 user 100 through a "view port", relative in time, along an "event horizon" 104. As will be described, a view port 
provides a perspective view of data contained in a "venue", wherein a venue may include in a plurality of resour- 
ces such as audio resources 102, video resources 95, etc. 

Referring now to Figure 4, the concept of venues is described in relation to three-dimensional space. The 
venue concept allows the user 100 to view data represented in alternative view ports of a venue. Relationships 

15 between a variety of resources are apparently based on the view which the user 100 chooses. As illustrated 
in Figure 4, a venue is a three-dimensional space in which object "elements" reside. An "element" is a three 
dimensional representation of a resource coupled to, for example, the network 44 depicted in Figure 2. Depend- 
ing on the view port which is chosen, the elements identify themselves in various ways. For example, in Figure 
4, element 110 may comprise a graphics resource, element 114 may represent an audio resource, element 

x 116 may represent a textual planning resource, and element 118 may represent a music or other audio 
resource. Viewing a venue through a top view port reveals the relative placement of the various elements in 
time, both the duration of time that an element represents (such as a film "take") and the relationship between 
the resources to a reference time, which may be a recording machine, finished film or other multi-media piece. 
Viewing the right side view port reveals versions of the elements, for example, the audio element 1 18 includes 

25 two versions, namely, a version 120 and a version 122, along with the relative time lengths of the versions. In 
addition, it should be noted that the user interface permits the use of multiple venues, wherein some or all of 
the venues may share the same time period, and each venue includes its own elements. 

For sake of example, assume that the elements illustrated in Figure 4 represent resources for the produc- 
tion of a soap opera. In this example, assume that a production crew photographed a variety of scenes, trans- 

30 ferred these scenes to video tape, and that these scenes comprise scene 1 , take 1 , etc. 

As shown in Figure 5, viewing an element through the front view port reveals the type of resource through 
the use of an icon label. For example, in the case of an audio resource, the icon label may comprise a graphic 
representation of a musical note. In addition, the various versions are illustrated, as in the case of the element 
118 in Figure 4, by darkened lines traversing the longitudinal length of the rectangular element image. With 

35 respect to Figure 5, the version may be activated, and thereby run, by placing a cursor on the screen of the 
display 50 over one of the activation buttons 130 on the element, and providing an activation signal, such as 
for example from a "mouse", to run that particular version of the resource. Returning to the present example 
of a soap opera production, the versions 120 and 122 of the audio element 118 as shown in Figure 4 and the 
versions 1 35 of the element 140 shown in Figure 5 may comprise musical options for use during the particular 

40 time associated by the length of the elements. Similarly, if the resource comprises scenes, then, as is common 
in the industry, each of these scenes may have associated time codes (such as SMPTE) which comprise the 
time in and out for each scene. Accordingly, by viewing the element representing the resource in terms of time, 
the time length of each scene would be represented on the element by the length of the particular version lines, 
for example the lines 1 35, or alternatively, by the duration lines 1 40 defining the beginning and end time of each 

45 scene of the resource. 

Referring now to Figure 6, the integrated user interface of this invention will be described. As shown in Fig- 
ure 6, the user interface includes a basic control frame 1 50 which in practice fills substantially all of the display 
screen of the display 50 as depicted in Figure 2. The control frame 150 is the primary context for interacting 
with the user interface and is, primarily, comprised of four control panels which surround a variety of subwin- 

50 dows. Three of the panels are visible to the user at all times, and the bottom panel (not shown) is displayed on 
an as needed basis. The mode specific action panel 152 comprises, in the displayed embodiment, a plurality 
of direct action buttons, or icons, which change with the program mode. As shown in the illustration of Figure 
6, the mode specific action panel 152 comprises a visual element 155, an aural element 158, a text element 
162, a graphic element 164, and a process element 166. Although buttons 155, 158, 162, 164 and 166 are 

55 illustrated in Figure 6, it will be appreciated that the mode specific action panel buttons/icons change to support 
and reflect the current activities of the user for a particular venue. 

The control frame 150 further includes a major mode panel 170, which comprises an edit button 172, a 
graphics button 174, an audio button 176, a plan button 180, and a set up button 182. It will be noted that 
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although throughout this description icons, buttons, and the like are described, that the reference to buttons, 
icons, etc. represents any class of displayed items which result in some executable action when chosen by a 
user. Therefore, although an edit button 172 is disclosed as part of the control frame 150, it will be appreciated 
to one skilled in the art that the edit button 172 may comprise an icon in the form of some edit feature or the 
5 like which achieves the same result. In the presently preferred embodiment, the buttons comprising the major 
mode panel 170 are always present for the user no matter which venue or other option is selected. 

In general, the major code panel 170 permits a user to access different venues than the venue currently 
displayed. The specific buttons/icons used in the major mode panel 170 are a function of the particular project 
in which the user interface is implemented. A menu bar panel 190 generally displays labels for pull down menus. 

10 Standard labels such as "organize", "modify", etc. are provided and are present no matter which venue or 
resource is accessed. Other context specific menu labels will be displayed, e.g. "element", "edit style", "utilities", 
"setup", etc are provided for specific applications. 

The application specific construction area 200 comprises an application area of the display for a selected 
program mode, and is available for subwindows, user views in the display of other elements of work such as 

15 time lines, scripts, scores, preview monitors, etc. As shown in Figure 6, the construction area 200 is designated 
as a resource manager. As will be described, a top view port 215 is also provided in the control frame 1 50. As 
previously illustrated in Figure 4, elements representing resources may be viewed and operated upon by 
appropriately selecting a view port. Although the control frame 150 displays objects in two dimensions, by 
appropriately selecting the view port, the elements may be viewed from all three dimensions. The elements 

20 directly represent the objects that make up a production, such as scripts, segments of video tape, score, scenes, 
director notes, sound tracks, etc., and are identified by an icon on the front face of the element as previously 
described with respect to Figure 5. Elements can be viewed from the front, side, or from the top, or in multiple 
views. As previously discussed with reference to the concept of "resources", the front view of the element dis- 
plays the icon label and the type of element may be determined from the icon as on its face (see Figure 5). 

25 Viewing an element from the top view port illustrates the relative length in time the elements may have. A view 
from a side port illustrates any different versions and their relative lengths. The element/resources may com- 
prise several individual elements, and may be bundled into a new compound element much like current users 
may group graphical elements using a graphics editor. 

Referring once again to Figure 6, the top view port 215 is used to position elements relative to time by plac- 

30 ing them on a time line. As will be described in more detail below, placing an element on a bar referred to as 
the "event horizon" 220 integrates the element into the overall time line for the production and results in the 
display of time data in the area identified in Figure 6 as 218. Moving an element from the construction region 
200 to the event horizon 220 results in the element being assigned a time assignment in the view port 215. It 
will be noted that the view port 215 corresponds to the top view port of an element in the three-dimensional 

35 representation of resources described with reference to Figure 4. 

Referring now to Figure 7, assume for sake of example that a user desires to define an element using the 
user interface. An element attribute box 250 is displayed by the computer 20 depicted in Figure 2 once a selec- 
tion identified as "element" 252 is chosen by a user. Although in the presently preferred embodiment, the ele- 
ment 252 comprises a pull down menu (not shown) having a plurality of items for selection, conceptually the 

40 element 252 is selected by a user through the use of a cursor control device (see for example, United States 
Patent Reissue ko. 32633). Although a cursor control device, such as a "mouse" may be used to selected the 
element 252, it will be appreciated that the actual selection may be made using a variety of display input devices 
known in the art. It will also be appreciated to one skilled in the art, that other mechanisms for selecting the 
element 252 the like are known. Accordingly, the particular mechanism for selecting functions, items, icons and 

45 the like within the control frame 150 will not be described in detail. 

Once the element 252 is selected, the computer 20 depicted in Figure 2 displays the element attribute box 
250 as shown in Figure 7. A user then either selects from a preprinted list of elements, or defines elements 
within the element attribute box 250, the nature of the resource required for the particular production. Examples 
of such element attributes which a user may select include, but are not limited to the following: 

50 

VI SUAL ELEMENT ATTRI BUTES 



User Supplied Element identification 
(e.g.: Video Tape Recorder, etc.) 
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SOURCE DEVICE 



Assignment of device: 

(e.g.: PI. P2, RECORD, etc.) 



IDENTIFICATION: 

Alpha-Numeric material 
Identification: e.g.: 
reel number, reel label, 
lab roll number, etc. 



SCENE/TAKE INFORMATION: 



Content Scene and Take identification. 
Scene Take file name. 



Time Code and type. 
Origin Time Code and type. 
User bit Time Code. 
User bits. Content. 
Frame Numbers. 
Edge Numbers 
Code Numbers 



Associated Text Notes 
For Reel L/or Content 



User assigned Priority levels 
for different versions 



PROCESSING PATH: 

Information on any previous 
Processing that applies to 
this material. 

(e.g: DUB LEVEL, COLOUR CORRECTION, ETC.) 



AUDIO ELEMENT ATTRTRUTRS 

55 
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LABEL: 

User supplied Element identification 

SOURCE DEVICE: 

Type and Assignment of device, 
(e.g. ATR, DISC, ETC.) 
PI, P2, RECORD, ETC. 



IDENTIFICATION 

Alpha-Numeric material 
identification 

(e.g. Reel number, Reel label etc. 
Sound Roll Number, Label). 



SCENE/TAKE § 

Content Scene and Take identification. 



Time Code and Type. 
Origin Time Code and type. 
User bit Time Code contents. 
Frame Numbers 



TRACKS 

Number and Numbers of Source Track 



Associated Text Notes. 
For Reel fc/or Content 



User assigned Priority levels 
for different versions 



PROCESSING PATH: 

Information on any previous 
Processing that applies to 
this material. 
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(e.g. Dub level, Equalization, etc) 
TEXT ELEMENT ATTRIBUTES 

LABEL: 

User supplied Element identification. 



Title of Text type and Document. 

(SCRIPT. 

OPENING TITLE, 

CHARACTER GENERATOR. ETC) 



The current Text revision level 
relevant previous revision information 



FILE TYPES: 



The Names and Types of files as 

the material exists in useable form. 



ASSOCIATED DATA: 



Any previous data files associated 
with creating the current file. 



PROCESSING PATH: 



Information on any previous 

Processing that applies to this material. 



GRAPHIC ELEMENT ATTRIBUTES 

LABEL: 

User supplied Element identification 



A user supplied description of 
the Graphic element 
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The current Graphic revision level 

and relevant previous revision information. 



FILE TYPES: 



The Names and Types of files as 

the material exists in useable form now. 



ASSOCIATED DATA: 



Any previous data files associated 
with creating the current file. 



PROCESSING PATH: 

Information on any previous 

Processing that applies to this material 



PROCESSING ELEMENT ATTRIBUTES 

LABEL: 

User Supplied Element identification. 

DEVICE IDENTIFICATION: 

Effects Device identification 

ASSIGNMENT PATH: 

Video and/or Audio routing assignments 
(e.g. CROSSPOINTS, KEY CHANNELS, ETC.) 



UP LOAD/DOWN LOAD: 

File input/output for created effects 
save and recall. 



Time Line Code and type. 
Effects durations. 
Effects Source codes. 
Effects Edge Numbers for 
optical printer outputs . 
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PROCESSING: 

Effects types, (e.g. CUTS, FADES, 
DISSOLVES, WIPES, KEYS. DME, ETC.) 



Once the element attributes have been defined, the computer 20 illustrated in Figure 2 utilizes appropriate 
network connections over the network 44 to the various resources, such as the VTR 52, music the synthesizer 
55, the Audio tape recorder 60, the Production switcher 62, etc. to access the resource via the user interface. 

w Accordingly, a direct connection via the computer 20 has been created between the user interface comprising 
the control frame 150 as displayed on the display 50, and the particular element/resource coupled through the 
network interface 42. Referring to Figure 7, within the top view port 215, time elements corresponding to the 
particular resource have additional information that is revealed in the time line associated with the top view port 
215. Tracking buses 265 provide additional information regarding the recording of audio channels 1 to 4, and 

15 a video channel 1. In practice, a source tape machine (not shown) supplies audio to a tape record machine 
wherein the channels are coupled to one another. It has been found that it is quite useful to display audio chan- 
nels in the time view port 215, in order to correlate the audio channel and time interval versus resource. 

Once the element attributes have been defined, an element representing a resource is created based upon 
those attributes, and displayed within the construction area 200 of the control frame 150. Referring now to Fig- 

20 ure 8, the user interface is illustrated wherein a plurality of elements identified as Record VTR 300, Scene I 
310, Scene II 312, Dissolve 314, Open Title 316, and Key 320 are shown. As illustrated in Figure 8, an element 
such as Record VTR 300 includes an icon image (for example, the planet Earth in Figure 8) which describes 
some aspect of the element for identification purposes. Viewing elements disposed in the construction area 
200 normally correspond to viewing a venue and associated resources through a front view port as shown pre- 

25 viously with respect to Figure 4. An element, for example, Record VTR 300, may be moved within the construc- 
tion region 200 at will by a user through the use of an appropriate command sequence, or by simply dragging 
the element around the construction area using a cursor control device such as a mouse. However, once an 
element such as Scene I 310 is brought, dragged, or otherwise manipulated downward to the event horizon 
220, the element is automatically given time significance which is represented along the time lines of the top 

30 view port 215. 

As illustrated in Figure 8, the event horizon 220 comprises a horizontal bar with arrows 221 and 222 at 
each of its opposite ends. By placing a cursor (not shown) over the arrows 221 or 222, and presenting the com- 
puter 20 with an activation signal, resource elements such as Scene I 310, Scene II 312, Dissolve 314, etc. 
may be moved left or right, respectively, and other elements may be viewed which are currently not visible on 
35 the event horizon 220 in the control frame 1 50. The use of the arrows 221 and 222 permits a user to scan through 
elements disposed on the event horizon 220 and view the elements not only in relative position, but in relative 
time. This view corresponds to that of a user 1 00 in Figure 3 scanning the resources in that Figure, such as 
special effects 90, editing 92, video 95, etc. In addition, it will be appreciated that the relative position of the 
element may be changed by simply "dragging" an element such as Scene I 310 off the event horizon 220, mov- 
40 ing other elements into that time slot along the event horizon 220 and replacing Scene I 310 at some other 
location along the event horizon 220. A redistribution of the element's relative position along the event horizon 
220 would correspond in Figure 4 to, for example, swapping the element 1 10 for the element 1 16 and vice versa. 

Once an element is placed upon the event horizon 220, position data relative in time to other elements is 
illustrated along the time lines of the top view port 21 5 as shown. Conceptually, the reader is directed to Figure 
45 4 which illustrates in three dimensions the placements of elements relative to one another in time. However, 
due to the limitations of the display 50 depicted in Figure 2, the time view port 215 is utilized to display time 
along the ± Y direction, with time being T G being at the lower portion of the display as illustrated in Figure 8. In 
addition, as shown in Figure 8, the number of versions, of, for example, Scene I 31 0, is also displayed as ver- 
sions 351 to 354. It will be appreciated by the reader that the display of an element such as Scene I 310 cor- 
se responds to the prior description of a resource having multiple versions which may be activated by selecting 
(for example, by placing a cursor over the version 354 and providing an activation signal) a version such that 
the version is "run" within the window of the Scene I 310. Accordingly, a user may view the entire version of 
Scene I which has been selected, within the icon window comprising the scene. In general, in the present and 
preferred embodiment, and throughout this specification, placing a cursor over an icon or other executable func- 
55 tion and "double clicking" using a cursor control device such that two consecutive signals are provided to the 
computer 20 depicted in Figure 2, executes the function which has been selected, and, more particularly, reve- 
als any attributes and/or contents of the icon. Double clicking on a time function such as the time block 400 for 
Scene I (Figure 8) may be configured such that time code (i.e. SMPTE) is displayed. More particularly, in the 
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present example, SMPTE time code for the beginning and end of each version within Scene I 310 may be dis- 
played within the time line view port 21 5. 

Referring now to Figure 9, the control frame 1 50 is illustrated in which a new Scene I 400 has been 
"dragged" upward into the construction area 200. As illustrated in Figure 9, once an element, in the present 

5 example Scene I 400, is moved off of the event horizon 220, timing information viewed through the top view 
port 215 corresponding to Scene I 400 is no longer displayed. Elements such as Scene II 404, Dissolve 406, 
Open Title 408 or key 410 may be repositioned along the event horizon 220 and/or modified in terms of time 
sequence as viewed through the top view port 215, relative to one another. Alternatively, and as shown in Figure 
9, the element attribute box 250 may be selected and the attributes of Scene I 400 may be modified, or an 

10 entirely new element may be defined, to replace Scene 1 400 along the event horizon 220. 

It will be appreciated that a user utilizing the interface defines elements in the construction area 200 by 
specifying attributes of the element in the element box 250. In addition, multiple elements may be created within 
the construction area 200. The created elements may then be selectively dragged to the event horizon 200 in 
an arrangement and order selected by the user. It will further be appreciated from the above discussion by one 

15 skilled in the art that the user interface permits the utilization of resources within the system illustrated in Figure 
2, permits selective modification of the resources, and through the use of the interface provides a consistent 
interface for the production of an audio visual work. The user interface through the use of the common control 
frame 1 50 as displayed on the display 50, allows artists, musicians, and other media professionals to create, 
modify, and rearrange resources comprising a production with flexibility heretofore unknown in the art. The con- 

20 cept of venues, and the ability to operate on resources in three dimensions, provide a user with flexibility not 
present in any prior user interface for computer display systems, we well as multi-media production systems 
known in the past. 

This embodiment of the invention has been described and illustrated with reference to the figures as applied 
to a display 50, and using input devices, such as the digital pad 36, the trackball 40 and the keyboard 38 as 

25 shown in Figure 2. However, numerous other display devices and input mechanisms may be used. For example, 
embodiments of this invention may be practised using what are known as "virtual reality" input devices, such 
as but notlimited to, adata input glove, a body glove input device, etc. In addition, embodiments of this invention 
may utilize eye goggle displays which are worn by a user and coupled to the computer display system via fibre 
optics, wires and the like. When embodiments of this invention utilizes a virtual reality system, the user interface 

30 would be viewed by a user through input goggles as being suspended in space. Interaction with the interface 
by the user may be done using an input glove or other virtual reality device worn by the user. Accordingly, it 
will be appreciated that the user interface is not limited to conventional input or display devices. The reader is 
referred to the following references for a further description of existing and proposed virtual reality systems. 
Computerised Reality Comes of Age , NASA Tech Briefs, page 10, August 1990 (Vol. 14, number 8); Iwata, 

35 Artificial Reality with Force -Feedback; Development of Desktop virtual Space with Compact Master mani- 
pulator , ACM SIGGRAPH, August 1990 (Volume 24. number 4); Nash, Our Man in Cyberspace Checks out 
Virtual Reality, Compterworld, October 15, 1990; Daviss, Grand Illusions , Discover, June 1990. 

At least preferred embodiments of this invention provide a multi-media user interface with may be utilized 
by a variety of multi-media professionals in the production of film or tape works. Unlike prior art window based 

40 display systems, a three-dimensional representation (known as an "element") of information to be manipulated 
is provided. Each element comprises a three dimensional representation of a "resource". A resource is a three 
dimensional object which may have data represented in either two or three-dimensional form. A window com- 
prises a "venue" which may be configured for specific activities, such as music production, special effects, 
scheduling and the like. However, the user interface shares common fundamental tools and the same data 

45 base, such that each media professional, such as a videotape editor, audio editor, producer, etc. may utilize 
the interface in a consistent manner. 

While this invention has been described in conjunction with a few specific embodiments identified in Figures 
1 to 9, it will be apparent to those skilled in the art that many alternatives, modifications and variations as may 
fall within the scope of the invention as disclosed. 

50 

Claims 

1. A computer controlled display apparatus including at least one central processing unit (CPU) (22), said 
55 CPU being coupled to a display (50) for displaying data, and user input means (36, 38, 40, 41 ), said display 

system being further coupled to a plurality of system resources (90, 92, 95, 102) having defined attributes, 
said display system comprising: 

means for generating a user interface for display by said display, said user interface including a 
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display of representations (300, 310, 312, 314, 316, 320) of resources coupled to said display system with 
which a user interacts through said user input means, said representations of said resources being 
arranged in an N dimensional venue which may be viewed using said user interface from a plurality of 
view ports (200, 215) such that viewing said representations of said resources from different view ports 
5 results in the display of different attributes of said resources, said representations of said resources being 

arranged in said venue such that each of said resources is disposed relative to one another in time and 
space within said venue; and 

manipulation means coupled to said user input means for selectively positioning said represen- 
tations of said resources within said venue. 

10 

2. Apparatus as claimed in claim 1 wherein said means for generating a user interface includes means coup- 
led to said CPU for generating and displaying a control frame (15) using said display to display selected 
view ports of a venue, said control frame including a plurality of command options (152, 190, 170) which 
may be selected by said user using said user input means. 

15 

3. Apparatus as claimed in claim 2 wherein said control frame further includes a event horizon bar (220), 
such that the placement of a representation of a resource on said bar results in timing data being displayed 
in said control frame. 

20 4. A computer controlled display system including at least one central processing unit (CPU), said CPU being 
coupled to display means for displaying data on a screen of a display, and user in put means, said display 
system being further coupled to a plurality of system resources having defined attributes, said display sys- 
tem comprising: 

means for generating a user interface on said screen of said display displaying representations of 
25 resources coupled to said display system with which a user interacts through said user input means, said 

means for generating a user interface including means coupled to said CPU for generating and displaying 
a control frame on said display, said control frame including a plurality of command options which may be 
selected by said user using said user input means, and further including a event horizon bar, such that 
the placement of a representation of a resource on said bar using said user input means results in pre- 
30 determined data being displayed in said control frame. 

5. Apparatus as claimed in claim 4 wherein said representations of said resources are arranged in an N 
dimensional venue which may be viewed using said interface from a plurality of view ports, such that view- 
ing said representations of said resources from different view ports results in the display on said screen 

35 of different attributes of said resources. 

6. Apparatus as claimed in claim 5 wherein said representations of said resources are arranged in said venue 
such that each of said resources is disposed relative to one another in time and space within said venue. 

40 7. Apparatus as claimed in claim 6 further including manipulation means coupled to said user input means 
for selectively positioning said representations of said resources within said venue. 

8. Apparatus as claimed in any one of claims 1, 5, 6 and 7 wherein said venue is three-dimensional and each 
of said representations of said resources is three dimensional. 

45 

9. Apparatus as claimed in claim 8 wherein said venue may be viewed from six view ports, namely, a top 
(215), left, right, rear, bottom and front (200) view port. 

10. Apparatus as claimed in claim 9 wherein viewing said venue from a top view port (21 5) reveals the relative 
so positions of each of said three dimensional representations of said resources relative in time to one 

another. 

11. Apparatus as claimed in claims 9 and 10 wherein viewing said venue from a front view port (200) reveals 
an icon identifying the type of resource each of said representations represent. 

55 

12. Apparatus as claimed in any one of claims 9, 10and 11 wherein viewing said venue from a side view port 
reveals versions (1 20, 122; 1 35) of said resource and the lengths of said versions relative to one another. 
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13. Apparatus as claimed in claim 12 wherein viewing said venue from a front view port further reveals version 
activation buttons (1 30; 350, 351 , 352, 353, 354), one of said buttons for each of said versions, such that 
said user, using said user input means may selectively activate said versions. 

5 14. Apparatus as claimed in claim 13 wherein in the event said user activates one of said versions, said version 
is executed within said representation of said resource, and may be viewed through said front view port 
of said venue on said display. 

15. Apparatus as claimed in any one of claims 2 to 14 wherein said control frame further includes a first area 
10 (200) for defining said attributes of said resources and displaying said representations of said resources 

once said attributes are defined. 

16. Apparatus as claimed in claim 15 wherein said first area displays said representations of said resources 
in a venue initially from a first view port. 

15 

1 7. Apparatus as claimed in claim 1 6 wherein said user may selectively change view ports from said first view 
port by selecting one of said command options. 

18. Apparatus as claimed in claims 15, 16 and 17 wherein said control frame further includes a second area 
20 (215) for displaying said top view port of said venue, such that timing data representing relative time 

associated with said resources is displayed in said second area. 

19. Apparatus as claimed in any one of claims 2 to 18 wherein selecting an Element command option results 
in the display of an element attribute box (250) for defining said resource attributes. 

25 

20. Apparatus as claimed in any one of claims 3 to 19 wherein said representations of said resources may 
be selectively placed on said event horizon bar, thereby altering the relative placement of said represen- 
tations in said venue in time and space. 

30 21. Apparatus as claimed in any one of claims 2 to 20 wherein said command options of said control frame 
further includes a plurality of mode specific action options (152) on the periphery of said control frame. 

22. Apparatus as claimed in any one of claims 2 to 21 wherein said command options of said control frame 
further includes a plurality of major mode options (17) on the periphery of said control frame. 

35 

23. Apparatus as claimed in claim 9 and 16 wherein said first view port initially displayed comprises the front 
view port. 

24. A method, in a computer controlled display system including at least one central processing unit (CPU), 
40 said CPU being coupled to a display for displaying data on a screen of said display, and user input means, 

said display system being further coupled to a plurality of system resources having defined attributes, of 
manipulating and displaying representations of said resources on said display comprising the steps of: 

providing a user interface on said screen of said display for displaying representations of resources 
coupled to said display system with which a user interacts through said user input means; 
45 generating and displaying a control frame on said display using interface generation means coupled 

to said CPU, said control frame including a plurality of command options which may be selected by said 
user using said user input means, and further including a event horizon bar, such that the placement of a 
representation of a resource on said bar using said user input means results in predetermined data being 
displayed in said control frame. 

50 

25. A method as claimed in claim 24 wherein said control frame further includes a first area for defining said 
attributes of said resources and displaying representations of said resources once said attributes are 
defined. 

55 26. A method as claimed in any one of claims 24 and 25 further including the step of selecting an Element 
command option and displaying an element attribute box for defining said resource attributes. 

27. A method as claimed in any one of claims 24, 25 and 26 wherein said command options of said control 
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frame further includes a plurality of mode specific action options on the periphery of said control frame. 

28. A method as claimed in any one of claims 24 to 27 wherein said representations of said resources are 
arranged in an N dimensional venue which may be viewed using said user interface from a plurality of 

5 view ports, such that viewing said representations of said resources from different view ports results in 

the display on said screen of different attributes of said resources. 

29. A method as claimed in claim 28 wherein said representations of said resources are arranged in said venue 
such that each of said resources is disposed relative to one another in time and space within said venue. 

10 

30. A method as claimed in any one of claims 28 and 29 wherein said venue is three dimensional and each 
of said representations of said resources is three dimensional. 

31. A method as claimed in claim 30 wherein viewing said venue from a top view port reveals the relative posi- 
15 tions of each of said three dimensional representations of said resources relative in time to one another. 

32. A method as claimed in any one of claims 30 and 31 wherein viewing said venue from a front view port 
reveals an icon identifying the type of resource each of said representations represent. 

20 33. A method as claimed in any one of claims 30, 31 and 32 wherein viewing said venue from a side view port 
reveals versions of said resource and the lengths of said versions relatives to ons another. 

34. A method as claimed in claim 33 wherein viewing said venue from a front view port further reveals version 
activation buttons, one of said buttons for each of said versions, such that said user, using said user input 

25 means may selectively activate said versions. 

35. A method as claimed in claim 34 further including the steps of said user activating one of said versions, 
said version then being executed within said representation of said resource, viewed through said front 
view port of said venue on said display. 

30 

36. A method as claimed in any one of claims 25 to 35 wherein said first area displays said representations 
of said resources in a venue initially from a first view port. 

37. A method as claimed in claim 36 further including the step of said user selectively changing view ports 
35 from said first view port by selecting one of said changing options. 

38. A method as claimed in any one of claims 30 to 35 wherein said control frame further includes a second 
area for displaying said top view port of said venue, such that timing data representing relative time 
associated with said resources is displayed in said second area. 

40 

39. A method as claimed in any one of claims 24 to 38 further including the step of selectively placing said 
representations of said resources on said event horizon, thereby altering the relative placement of said 
representations in said venue in time and space. 

45 40. A method as claimed in any one of claim 24 to 39 wherein said command options of said control frame 
includes a plurality of major mode options on the periphery of said control frame. 

41. A method as claimed in claims 30 and 36 wherein said first view port initially displayed comprises the front 
view port. 

50 

42. A resource control apparatus comprising: 

a display (50), 

user interface generation means (26) for displaying on said display representations (300, 310, 312, 
314, 416, 320) of a plurality of resources (90, 92, 95, 102) that may be coupled to said resource control 
55 apparatus, said representations being displayable by said user interface generation means from a plurality 

of view ports such that viewing said representations from different view ports (200, 215) results in the dis- 
play of different attributes of said resources in a manner consistent with said representations being of N 
dimensional elements (1010, 114, 116, 188) disposed within an N dimensional venue, and 
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manipulation means responsive to a user input means (36, 38, 40, 41) for selectively positioning 
said representations within said venue. 
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(&4) Computer controlled display apparatus and method. 

(57) Apparatus and methods are described to pro- 
vide a multi-dimensional user interface for use 
in audio visual production. A display system 
including a central processing unit (CPU) (22) is 
coupled through appropriate input/output (I/O) 
circuitry (32) to input devices, such as a 
keyboard (38), a digital pad (36) and/or a track 
ball (40) as well as a display device (50). The 
CPU is further coupled to a hard disk drive (30) 
for the storage of programs and data, and is ' c 
also coupled to a network through which the 
CPU may communicate with a variety of system 
resource devices such as editors, music synth- 
esizers (55), graphics generators, scheduling 
resources, audio enhancement resources, etc. 
A user viewing the interface on the display may 
utilize one of the input devices, such as by way 
of example, the keyboard, to select, incorporate 
or otherwise integrate the various system re- 
sources to develop a unified multi-media pro- 
duction. The user interface includes a control 
frame (150) which in practice substantially fills 
all of the display screen of the display and is 
consistent for all user applications. The control 
frame is comprised of control panels (152, 190, 
170) which surround a variety of subwindows 
(200, 215) and acts as a consistent control area 
for all users of the interface. Once defined, 
elements may be selectively placed on an event 
horizon bar (220) in the control frame. The 
placement of an element on the event horizon 
results in the display of timing data for the 
element, relative to other elements on the event 
horizon. 
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